FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

摘要

The FlowNet demonstrated that optical flow estimation can be cast as alearning problem. However, the state of the art with regard to the quality ofthe flow has still been defined by traditional methods. Particularly on smalldisplacements and real-world data, FlowNet cannot compete with variationalmethods. In this paper, we advance the concept of end-to-end learning ofoptical flow and make it work really well. The large improvements in qualityand speed are caused by three major contributions: first, we focus on thetraining data and show that the schedule of presenting data during training isvery important. Second, we develop a stacked architecture that includes warpingof the second image with intermediate optical flow. Third, we elaborate onsmall displacements by introducing a sub-network specializing on small motions.FlowNet 2.0 is only marginally slower than the original FlowNet but decreasesthe estimation error by more than 50%. It performs on par with state-of-the-artmethods, while running at interactive frame rates. Moreover, we present fastervariants that allow optical flow computation at up to 140fps with accuracymatching the original FlowNet.